Evaluation of Automatically Generated Pronoun Reference Questions
نویسندگان
چکیده
This study provides a detailed analysis of evaluation of English pronoun reference questions which are created automatically by machine. Pronoun reference questions are multiple choice questions that ask test takers to choose an antecedent of a target pronoun in a reading passage from four options. The evaluation was performed from two perspectives: the perspective of English teachers and that of English learners. Item analysis suggests that machinegenerated questions achieve comparable quality with human-made questions. Correlation analysis revealed a strong correlation between the scores of machinegenerated questions and that of humanmade questions.
منابع مشابه
Automatic Generation of English Reference Question by Utilising Nonrestrictive Relative Clause
This study presents a novel method of automatic English reference question generation. The English reference question is a multiple choice question which is comprised of a reading passage, a target pronoun, a correct answer (an antecedent of the target pronoun), and three distractors. The reading passage is generated by transforming human made passages using the proposed sentence splitting tech...
متن کاملA Graphical Pronoun Analysis Tool for the PROTEST Pronoun Evaluation Test Suite
We present a graphical pronoun analysis tool and a set of guidelines for manual evaluation to be used with the PROTEST pronoun test suite for machine translation (MT). The tool provides a means for researchers to evaluate the performance of their MT systems and browse individual pronoun translations. MT systems may be evaluated automatically by comparing the translation of the test suite pronou...
متن کاملA study of the reference of pronoun ‘this’ in “All praise is due to Allah who guided us to this and we would not have found the way had it not been that Allah had guided us”
We encounter with some pronouns in the Holy Quran that are with no reference. Of course this makes mind search for their reference with regard to the verses’ meaning and intention, otherwise it views as subjective interpretation of Quranic text. One of these cases is the pronoun ‘this’ in (7: 43) in which anyone who is in Paradise says: “All praise is due to Allah who gu...
متن کاملValidation of an Automatic Metric for the Accuracy of Pronoun Translation (APT)
In this paper, we define and assess a reference-based metric to evaluate the accuracy of pronoun translation (APT). The metric automatically aligns a candidate and a reference translation using GIZA++ augmented with specific heuristics, and then counts the number of identical or different pronouns, with provision for legitimate variations and omitted pronouns. All counts are then combined into ...
متن کاملPROTEST: A Test Suite for Evaluating Pronouns in Machine Translation
We present PROTEST, a test suite for the evaluation of pronoun translation by MT systems. The test suite comprises 250 hand-selected pronoun tokens and an automatic evaluation method which compares the translations of pronouns in MT output with those in the reference translation. Pronoun translations that do not match the reference are referred for manual evaluation. PROTEST is designed to supp...
متن کامل